Seamless Video Pipeline Transition Between WiFi and Cellular Connections for Real-Time Applications on Mobile Devices

ABSTRACT

Performing a real-time application on a mobile device, involving communication of audio/video packets with a remote device. The mobile device may initially communicate the audio/video packets on a first communication channel with the remote device. During the real-time communication, the mobile device may determine if no packets have been received by the mobile device from the remote device for a first threshold period of time. If no packets have been received by the mobile device from the remote device for the first threshold period of time, the mobile device may establish a second communication channel for transmission of the audio/video packets with the remote device. In response to using the second communication channel, the mobile device may modify a resolution or bit rate of the audio/video packets transmitted to the remote device.

PRIORITY INFORMATION

The present application is a continuation of U.S. patent applicationSer. No. 14/338,734 titled “Seamless Video Pipeline Transition BetweenWiFi and Cellular Connections for Real-Time Applications on MobileDevices” and filed Jul. 23, 2014, whose inventors are Yan Yang, XiasongZhou, Hyeonkuk Jeong, Joe S. Abuan, Gobind Johar, and Yichao Shen, whichclaims benefit of priority to provisional patent application No.62/005,263, entitled “Seamless Video Pipeline Transition Between WiFiand Cellular Connections for Real-Time Applications on Mobile Devices”,filed on May 30, 2014, whose inventors are Yan Yang, Xiasong Zhou,Hyeonkuk Jeong, Joe S. Abuan, Gobind Johar, and Yichao Shen, and whichare all hereby incorporated by reference in their entirety as thoughfully and completely set forth herein.

FIELD OF THE INVENTION

The present application relates to wireless devices, and moreparticularly to providing an improved user experience for real-timeapplications executing on a mobile device using redundant transmissionchannels.

DESCRIPTION OF THE RELATED ART

Wireless communication systems are rapidly growing in usage. Further,wireless communication technology has evolved from voice-onlycommunications to also include the transmission of data, such asInternet and multimedia content. Some popular applications includevideoconferencing (e.g., such as FaceTime™ from Apple) as well as mediastreaming. However, in many instances, the transmission of video contentover a wireless communication channel such as WiFi suffers from channeldegradation issues. Therefore, improvements are desired in wirelesscommunication.

SUMMARY OF THE INVENTION

Embodiments described herein relate to a method for performing areal-time application on a mobile device, involving communication ofaudio/video packets with a remote device. The mobile device mayinitially communicate the audio/video packets on a first communicationchannel with the remote device. During the real-time communication, themobile device may measure downlink channel quality of the first channel,e.g., by determining if no packets have been received by the mobiledevice from the remote device for a first threshold period of time. Ifno packets have been received by the mobile device from the remotedevice for the first threshold period of time, the mobile device mayestablish a second communication channel for transmission of theaudio/video packets with the remote device.

In one embodiment, during the real-time application, the mobile devicemay capture first images from a camera of the mobile device using firstcamera capture settings having a first resolution. The mobile device maythen communicate first multimedia packets, corresponding to the firstimages, on a first communication channel during a video session. Inresponse to transitioning to communication of multimedia packets on asecond communication channel during the video session, the mobile devicemay change the camera capture settings of the camera to second cameracapture settings having a second resolution that is different from thefirst resolution. Accordingly, the mobile device may communicate secondmultimedia packets, corresponding to the second images, on the secondcommunication channel during the video session.

This Summary is provided for purposes of summarizing some exemplaryembodiments to provide a basic understanding of aspects of the subjectmatter described herein. Accordingly, the above-described features aremerely examples and should not be construed to narrow the scope orspirit of the subject matter described herein in any way. Otherfeatures, aspects, and advantages of the subject matter described hereinwill become apparent from the following Detailed Description, Figures,and Claims.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the embodiments is considered inconjunction with the following drawings.

FIG. 1 illustrates an example mobile device, a smart phone, according toone embodiment;

FIG. 2 illustrates another example mobile device, a table computer,according to one embodiment;

FIG. 3 is an example block diagram of a mobile device, according to oneembodiment;

FIG. 4 illustrates an example wireless communication system where amobile device communicates using two different channels associated withdifferent RATs, respectively, according to one embodiment

FIG. 5 is a flowchart diagram illustrating an exemplary method for usinga plurality of channels for a real-time application, according to oneembodiment;

FIG. 6 is a flowchart diagram illustrating an exemplary method fortransitioning from dual channel transmission to single channeltransmission, according to one embodiment;

FIG. 7 is a flowchart diagram illustrating an exemplary method formodifying settings based on changes in channel transmission;

FIG. 8 is a flowchart diagram illustrating an exemplary method fordynamic modification of camera capture settings;

FIG. 9 is a flowchart diagram illustrating an exemplary method fordynamic modification of camera capture settings and video encoder;

FIG. 10 is a flowchart diagram illustrating an exemplary method forusing I frames to notify the remote device of changed settings;

FIG. 11 is a flowchart diagram illustrating an exemplary method fortransmitting different amounts of data over primary and secondarychannels; and

FIG. 12 is a flowchart diagram illustrating an exemplary method fortransmitting varying amounts of droppable frames over primary andsecondary channels.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE EMBODIMENTS Acronyms

The following acronyms are used in the present disclosure.

WiFi: Wireless Fidelity

LTE: Long Term Evolution

RAT: Radio Access Technology

TX: Transmit

RX: Receive

TERMS

The following is a glossary of terms used in the present application:

Memory Medium—Any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks, or tape device; a computer system memoryor random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, RambusRAM, etc.; a non-volatile memory such as a Flash, magnetic media, e.g.,a hard drive, or optical storage; registers, or other similar types ofmemory elements, etc. The memory medium may include other types ofmemory as well or combinations thereof. In addition, the memory mediummay be located in a first computer system in which the programs areexecuted, or may be located in a second different computer system whichconnects to the first computer system over a network, such as theInternet. In the latter instance, the second computer system may provideprogram instructions to the first computer for execution. The term“memory medium” may include two or more memory mediums which may residein different locations, e.g., in different computer systems that areconnected over a network. The memory medium may store programinstructions (e.g., embodied as computer programs) that may be executedby one or more processors.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), personal communication device, smart phone, televisionsystem, grid computing system, or other device or combinations ofdevices. In general, the term “computer system” can be broadly definedto encompass any device (or combination of devices) having at least oneprocessor that executes instructions from a memory medium.

Mobile Device—any of various types of computer systems devices which aremobile or portable and which performs wireless communications using bothcellular communication and WiFi communication. Examples of mobiledevices include mobile telephones or smart phones (e.g., iPhone™,Android™-based phones), and tablet computers such as iPad, SamsungGalaxy, etc. Various other types of devices would fall into thiscategory if they include both cellular and WiFi communicationcapabilities, such as laptop computers (e.g., MacBook), portable gamingdevices (e.g., Nintendo DS™ PlayStation Portable™, Gameboy Advance™,iPhone™), portable Internet devices, and other handheld devices, as wellas wearable devices such as wrist-watches, headphones, pendants,earpieces, etc. In general, the term “mobile device” can be broadlydefined to encompass any electronic, computing, and/ortelecommunications device (or combination of devices) which is easilytransported by a user and capable of wireless communication using bothWiFi and cellular communication.

Cellular—The term “cellular” has the full breadth of its ordinarymeaning, and at least includes a wireless communication network or RATdistributed over land areas called “cells” and is used by virtually allmobile telephones today.

Cellular RAT—The term “cellular RAT” has the full breadth of itsordinary meaning, and at least includes any of the present or futureradio access technologies used to communication on a cellular network,such as Global System for Mobile Communications (GSM), Universal MobileTelecommunications System (UMTS), Code Division Multiple Access (CDMA)(e.g., CDMA2000 1×RTT or other CDMA radio access technologies), LongTerm Evolution (LTE), LTE Advanced, and other similar or future cellularRATs, including 3G, 4G, 5G, 6G, etc., cellular RATs.

Cellular Base Station—The term “Cellular Base Station” has the fullbreadth of its ordinary meaning, and at least includes a wirelesscellular communication station installed at a fixed location and used tocommunicate as part of a cellular network or cellular radio accesstechnology (RAT).

WiFi—The term “WiFi” has the full breadth of its ordinary meaning, andat least includes a wireless communication network or RAT that isserviced by wireless LAN (WLAN) access points and which providesconnectivity through these access points to the Internet. Most modernWiFi networks (or WLAN networks) are based on IEEE 802.11 standards andare marketed under the name “WiFi”. A WiFi (WLAN) network is differentfrom a cellular network.

Processing Element—refers to various elements or combinations ofelements. Processing elements include, for example, circuits such as anASIC (Application Specific Integrated Circuit), portions or circuits ofindividual processor cores, entire processor cores, individualprocessors, programmable hardware devices such as a field programmablegate array (FPGA), and/or larger portions of systems that includemultiple processors.

Automatically—refers to an action or operation performed by a computersystem (e.g., software executed by the computer system) or device (e.g.,circuitry, programmable hardware elements, ASICs, etc.), without userinput directly specifying or performing the action or operation. Thusthe term “automatically” is in contrast to an operation being manuallyperformed or specified by the user, where the user provides input todirectly perform the operation. An automatic procedure may be initiatedby input provided by the user, but the subsequent actions that areperformed “automatically” are not specified by the user, i.e., are notperformed “manually”, where the user specifies each action to perform.For example, a user filling out an electronic form by selecting eachfield and providing input specifying information (e.g., by typinginformation, selecting check boxes, radio selections, etc.) is fillingout the form manually, even though the computer system must update theform in response to the user actions. The form may be automaticallyfilled out by the computer system where the computer system (e.g.,software executing on the computer system) analyzes the fields of theform and fills in the form without any user input specifying the answersto the fields. As indicated above, the user may invoke the automaticfilling of the form, but is not involved in the actual filling of theform (e.g., the user is not manually specifying answers to fields butrather they are being automatically completed). The presentspecification provides various examples of operations beingautomatically performed in response to actions the user has taken.

FIGS. 1 and 2—Mobile Device

FIG. 1 illustrates an example mobile device 106A according to oneembodiment, in this case a telephone device, such as a smart phone. Theterm mobile device 106 may be any of various devices as defined above.FIG. 2 illustrates another example of mobile device 106B according toone embodiment, in this case a tablet device, such as a tablet computer,e.g., an iPad™, Samsung Galaxy™, and similar devices).

Mobile device 106 may include a housing 12 which may be constructed fromany of various materials. Mobile device 106 may have a display 14, whichmay be a touch screen that incorporates capacitive touch electrodes.Display 14 may be based on any of various display technologies. Thehousing 12 of the mobile device 106 may contain or comprise openings forany of various elements, such as home button 16, speaker port 18, andother elements (not shown), such as microphone, data port, and possiblyvarious other types of buttons, e.g., volume buttons, ringer button,etc.

The mobile device 106 may support multiple radio access technologies(RATs). For example, mobile device 106 may be configured to communicateusing any of various cellular RATs. The mobile device 106 may also beconfigured to support at least one WiFi RAT, also known as Wireless LAN(WLAN). Various different or other RATs may be supported as desired.

The mobile device 106 may comprise one or more antennas. The mobiledevice 106 may also comprise any of various radio configurations, suchas various combinations of one or more transmitter chains (TX chains)and one or more receiver chains (RX chains). For example, the mobiledevice 106 may comprise a radio that supports two or more RATs. Theradio may comprise a single TX (transmit) chain and a single RX(receive) chain. Alternatively, the radio may comprise a single TX chainand two RX chains that operate on the same frequency. In anotherembodiment, the UE 106 comprises two or more radios, i.e., two or moreTX/RX chains (two or more TX chains and two or more RX chains).

FIG. 3—Mobile Device Block Diagram

FIG. 3 illustrates an example simplified block diagram of a mobiledevice 106. As shown, the mobile device 106 may include a system on chip(SOC) 400, which may include portions for various purposes. The SOC 400may be coupled to various other circuits of the mobile device 106. Forexample, the mobile device 106 may include various types of memory(e.g., including NAND flash 410), a connector interface 420 (e.g., forcoupling to a computer system, dock, charging station, etc.), thedisplay 460, cellular communication circuitry 430 such as for LTE, GSM,etc., and short range wireless communication circuitry 429 (e.g.,Bluetooth™ and WiFi circuitry). The mobile device 106 may furthercomprise one or more smart cards 310 that comprise SIM (SubscriberIdentity Module) functionality, such as one or more UICC(s) (UniversalIntegrated Circuit Card(s)) cards 310. The cellular communicationcircuitry 430 may couple to one or more antennas, preferably twoantennas 435 and 436 as shown. The short range wireless communicationcircuitry 429 may also couple to one or both of the antennas 435 and 436(this connectivity is not shown for ease of illustration).

As shown, the SOC 400 may include processor(s) 402 which may executeprogram instructions for the UE 106 and display circuitry 404 which mayperform graphics processing and provide display signals to the display460. The processor(s) 402 may also be coupled to memory management unit(MMU) 440, which may be configured to receive addresses from theprocessor(s) 402 and translate those addresses to locations in memory(e.g., memory 406, read only memory (ROM) 450, NAND flash memory 410)and/or to other circuits or devices, such as the display circuitry 404,cellular communication circuitry 430, short range wireless communicationcircuitry 429, connector I/F 420, and/or display 460. The MMU 440 may beconfigured to perform memory protection and page table translation orset up. In some embodiments, the MMU 440 may be included as a portion ofthe processor(s) 402.

In one embodiment, as noted above, the mobile device 106 comprises atleast one smart card 310, such as a UICC 310, which executes one or moreSubscriber Identity Module (SIM) applications and/or otherwise implementSIM functionality. The at least one smart card 310 may be only a singlesmart card 310, or the UE 106 may comprise two or more smart cards 310.Each smart card 310 may be embedded, e.g., may be soldered onto acircuit board in the UE 106, or each smart card 310 may be implementedas a removable smart card, or any combination thereof. Any of variousother SIM configurations are also contemplated.

As noted above, the mobile device 106 may be configured to communicatewirelessly using multiple radio access technologies (RATs). The mobiledevice 106 may be configured to communicate according to a WiFi RATand/or one or more cellular RATs, e.g., such as communicating on bothWiFi and cellular at the same time. For example, the mobile device 106may be communicating on a primary communication channel (such as WiFi),and in response to detected degradation of the primary communicationchannel may establish a secondary communication channel (such as oncellular). The mobile device 106 may operate to dynamically establishand/or remove different primary and/or secondary communication channelsas needed, e.g., to provide the best user experience while attempting tominimize cost.

As described herein, the mobile device 106 may include hardware andsoftware components for implementing the above features forestablishing, removing, and/or adjusting primary and secondarycommunication channels, as well as the various other techniquesdescribed herein. The processor 402 of the UE device 106 may beconfigured to implement part or all of the features described herein,e.g., by executing program instructions stored on a memory medium (e.g.,a non-transitory computer-readable memory medium). Alternatively (or inaddition), processor 402 may be configured as a programmable hardwareelement, such as an FPGA (Field Programmable Gate Array), or as an ASIC(Application Specific Integrated Circuit). Alternatively (or inaddition) the processor 402 of the UE device 106, in conjunction withone or more of the other components 400, 404, 406, 410, 420, 430, 435,440, 450, 460 may be configured to implement part or all of the featuresdescribed herein.

FIG. 4—Communication System

FIG. 4 illustrates an exemplary (and simplified) wireless communicationscenario involving multiple communication systems. It is noted that thesystem of FIG. 4 is merely one example of a possible system, andembodiments may be implemented in any of various systems, as desired.

As shown, the exemplary wireless communication system includes a mobiledevice 106 communicating over a primary communication channel 142. Inthis example, the primary communication channel is a WiFi network RAT,whereby the mobile device 106 communicates through a WiFi access pointas shown. The mobile device 106 may also establish a secondarycommunication channel 144, e.g., in response to channel conditions ofthe primary communication channel 142, as discussed in more detailbelow. In this example, the secondary communication channel is acellular network RAT, whereby the mobile device 106 communicates througha cellular base station as shown. The mobile device 106 may communicateover one or both of the primary communication channel 142 and thesecondary communication channel 144 to communicate with a remote device154, which is also discussed in more detail below. Additionally, notethat the RATs shown in FIG. 4 are exemplary only, and that othercombinations are envisioned, such as using multiple different cellularRATs, e.g., instead of or in addition to the WiFi RAT.

Note that FIG. 4 only illustrates one side of the communicationscenario, that is, the side from the mobile device 106 to the network152 (e.g., the Internet). The remote device 154 may similarly have oneor more paths to the network 152 from its point of view, e.g., includingWiFi and/or cellular networks, as desired. In one embodiment, a“channel” may be considered any end-to-end communication path betweenthe mobile device 106 and the remote device 154 (e.g., and may becharacterized or specified by pairs of addresses, e.g., IP address andport number combinations for the mobile device 106 and the remote device154). For example, a primary channel (e.g., the channel 142) may involvethe mobile communication device 106 communicating via WiFi to network152 and the remote device 154 communicating via its own WiFi to thenetwork 152. The secondary channel (e.g., the channel 144) may involvethe mobile communication device 106 communicating via a cellular RAT tothe network 152 and the remote device 154 communicating via its owncellular RAT to the network 152. However, while these examples show amirrored path on each side, the remote device 154 may use any RAT foreach channel, as appropriate. For example, the remote device 154 may useWiFi communication for both the primary channel and the secondarychannel. Alternatively, the remote device 154 may use a cellular RAT forthe primary channel and WiFi for the secondary channel. In someembodiments, the “channel” may be at an application level, althoughother levels are also envisioned, such as lower levels.

Establishing and Using Multiple Channels for Real-Time Applications

As mentioned previously some real-time applications on mobile devicessuffer from a degraded transmission channel while performing thereal-time applications. The real-time applications may includebidirectional real-time applications such as conferencing orvideoconferencing, or may include unidirectional applications, where theflow of traffic is almost solely in one direction (e.g., except forfeedback channels or data regarding the application), such as audio orvideo streaming. As an example of channel degradation during a real-timeapplication, a video conferencing session using WiFi can become unusableand may even be abruptly dropped when WiFi signal degrades, which has asignificant impact on the user experience of the real-time application.Therefore, it is important for real-time applications to minimize userimpact caused by a degraded transmission channel. In order to addressthese and other issues, various embodiments may be implemented.

For example, in one embodiment, an algorithm to perform fast detectionof degraded transmission channels on one or both of uplink and downlinkmay be implemented. Naive approaches may use packet loss, RTT (“roundtrip time”), or other similar measures to determine link quality.However, while these metrics may be useful for some purposes (e.g., andmay be used to augment various methods described herein), they maygenerally be slow, inaccurate, and cannot distinguish between uplink anddownlink. Because link quality may degrade from either downlink oruplink, or both, and it may typically take a long time to determine afailure on the channel using end-to-end metrics, a different approachmay be used. Different amounts of quality degradation can occur on thedownlink and uplink channels, and the quality degradation may occur atdifferent at different speeds on the two links. Therefore differentdetection algorithms may be applied herein to the downlink and uplinkchannels in order to detect link quality degradation on the differentlinks for fast detection.

For example, to determine link quality for the local downlink (e.g.,from the access point or base station to the mobile device) or for theremote uplink (e.g., from the remote device to its access point or basestation), the mobile device may determine if no packets have beenreceived from the remote device for a first threshold of time (e.g., a“No-Remote” condition has occurred). In one embodiment, the first timethreshold may be approximately three seconds, although the firstthreshold may vary, e.g., plus or minus two seconds or less. Uponreaching this condition, the mobile device may transmit a messageindicating that the condition has been reached (e.g., by sending a“No-Remote” message to the remote device). Alternatively, the mobiledevice may periodically send such a message indicating the length oftime since the mobile device has received a packet from the remotedevice, which the remote device can then use for its own algorithms,such as in the example in the following paragraph. One way to implementthis feedback is to utilize the RTP Control (RTCP) protocol, where “RTP”stands for Real-time Transport Protocol. To reduce message overhead,such information can be incorporated within, or “piggybacked” on,existing audio/video RTP packets. With RTP packets being sent at fixedintervals, further saving can be achieved by only using a bit-flag toindicate if a packet was received or not, instead of explicitly puttingtime information into the RTP packets.

Similarly, to determine link quality for the local uplink (e.g., fromthe mobile device to the access point or base station) or for the remotedownlink (e.g., from the remote access point or base station to theremote device), the mobile device may determine if no packets have beenreceived by the remote device form the mobile device for a secondthreshold of time (e.g., a “Remote-No-Remote” condition has occurred).In this case, the remote device may monitor for incoming packetstransmit a message to the mobile device indicating the amount of timeand/or that the amount of time has exceeded a threshold (e.g., theremote device may transmit a “No-Remote” message to the mobile device,indicating that the remote device has not received a packet from themobile for an amount of time or that its “No-Remote” condition has beentriggered). In some embodiments, the first and second thresholds may bethe same. Alternatively, the first and second thresholds may bedifferent. For example, the first threshold may be longer than the firstthreshold, e.g., in the example above, the first threshold may be threeseconds, but the second threshold may be two seconds. Similar to above,while the example second threshold may be approximately two seconds, itmay be other values, e.g., varying by plus or minus a second or more, asdesired.

In response to one or both of the conditions above, a redundant channelmay be established for the real-time application. For example, if theprocesses above indicate a loss of link quality for the channel (e.g.,one or both of the time thresholds have been exceeded), then a processmay be used to establish an additional channel between the mobile deviceand the remote device. For example, if the primary channel was a WiFi toWiFi channel (where both the mobile device and the remote device useWiFi for communicating using the real-time application), then thesecondary channel may be a cellular to cellular channel (where both themobile device and the remote device use a respective cellular RAT forcommunicating using the real-time application). Similarly, if theprimary channel was cellular to cellular, then the secondary channel maybe WiFi to WiFi. As another example, if the primary channel was WiFi tocellular, then the secondary channel may be cellular to WiFi. Similarly,if the primary channel was cellular to WiFi, then the secondary channelmay be WiFi to cellular. If one side only has one RAT available (e.g.,the remote device), then the primary channel may be switched from WiFito cellular while the remote device uses the only RAT available.

In one embodiment, rather than establishing the secondary channel inresponse to these conditions, the secondary channels may have beenpreviously negotiated or established, and the conditions may be used todetermine that the secondary channel should begin to be used. Forexample, the secondary channel may have already been established, butnot used, until the one or more of the thresholds have been met. Then,in one embodiment, packets (e.g., audio and/or video packets) of thereal-time application may be transmitted over both the primary and thesecondary channel concurrently, in a redundant fashion.

Transmitting data in dual channels can incur extra data charges to auser. For example, cellular carriers generally charge for data usageover a cellular connection. In some cases, transmission of data over aWiFi connection through a carrier's network can also result in charges.Hence, it is desirable to minimize the usage of dual channels totransmit audio/video data in order to reduce extra or unnecessary datausage charges. Therefore, once both channels are being used for thereal-time application, an algorithm may be used to determine if and/orwhen to return to using a single channel (e.g., either the primarychannel or the secondary channel or a further channel, depending onconditions). This algorithm may operate to determine when a singlechannel provides sufficient quality such that dual channels are nolonger necessary.

Note that embodiments for transitioning from dual channel transmissionto single channel transmission may be independent of the embodiments fortransitioning from single channel transmission to dual channeltransmission. For example, a different method than above may be used totransition from single to dual channel transmission and the followingembodiments may be similarly implemented. Similarly, a different methodthan below for transitioning from dual channel transmission to singlechannel transmission in combination with the transition from singlechannel to dual channel transmission. Of course, both sets ofembodiments may be used together in the same implementation, as desired.

Similar to above, naive approaches can measure RTT, PLR (“packet lossratio”), etc. for performing this transition. However, while thesemetrics may be useful for some purposes (e.g., and may be used toaugment various methods described herein), they may generally be slow,inaccurate, and cannot distinguish between uplink and downlink. Further,these naive approaches do not directly correlate networking relatedmetrics to the actual user experience for an audio/video conferencingapplication.

In one embodiment, the mobile device may measure the percentage ofreceived packets that are actually used by the real-time application,e.g., what percentage of the audio/video playback from audio/videodecoder is used for each channel. In particular, during dual channeltransmission, a same packet may be transmitted over both channels.However, there may be different latencies or issues associated with eachchannel, and the receiving device may receive the packets at differenttimes, or may not receive them at all if there are channel issues. Thus,the receiving device (which may be the mobile device or the remotedevice) may use the first successfully received packet of the redundantpackets and discard the other.

For example, an audio receiver in a video conferencing applicationnormally maintains a jitter buffer to ensure smooth audio playback. Herethe term “jitter” refers to variation in latency in the network, or morespecifically the variability of the packet latency as measured over timeacross the network. A normal jitter buffer implementation usuallydiscards duplicate or late arrived packets. Duplicate packets can bedetected by comparing sequence numbers in the packets, e.g., the RTPsequence number. The discarded packets are generally not used for audioplayback by the audio receiver. In a similar manner, when dual channelsare being used as described herein, duplicate or redundant packets maybe discarded and hence not used in a similar manner.

In this embodiment, the receiving device may then keep track of thepercentage of packets used for each channel. For example, if the primarychannel is very poor and most of the packets are not successfullytransmitted, only 15% of the packets from the primary channel may beused, in which case, the primary channel may no longer be necessary. Onthe other hand, if most are successful, then perhaps 90% or more of thepackets may come from the primary channel, in which case the secondarychannel may no longer be necessary.

Thus, in one embodiment, the percentage of packets used may be comparedto a threshold, e.g., for the primary channel, to determine whether tocontinue to perform dual channel transmission or transition to singlechannel transmission using the primary or secondary channel. Forexample, the threshold may be 70%, 80%, 90%, or 95% for switching tosingle channel transmission. As a specific example, if 88% of thepackets are used for the primary channel, and the threshold is 90%, thendual channel transmission may be continued. However, if the percentageincreases to 92%, then a transition to single channel transmission usingthe primary channel may be effected. Similarly, if the primary channelis used only 23% of the time and the threshold is 75%, then thesecondary channel may be used as the single channel, if desired. Notethat the threshold may be expressed as the percentage to exceed foreither of the channels or as the lower threshold to discontinue use ofone of the channels. For example, the threshold could be expressed asneeding to exceed 75% to transition to only using that channel, or asneeding to fall below 25% to transition to only using the other channel.Since there are only two channels in this embodiment, these areequivalent. For more than two channels, either embodiment may be used.

In one embodiment, this algorithm may be expressed as turning offtransmission over redundant channel if the primary channel shows goodpercentage consistently and switching to the secondary (or redundant)channel if the primary channel shows poor percentage consistently. Ineffect, the latter results in upgrading the secondary channel to theprimary channel. In order to avoid switching back and forth betweensingle and dual channel transmission too often or too frequently,various logic or algorithms may be used. For example, various timingthresholds or other hysteresis algorithms may be used.

Alternatively, or additionally, the method may use various quality ofservice measures to determine if a single channel could be used. Forexample, in cases where either channel is acceptable, and so thepercentage threshold may not be particularly useful (e.g., where bothare equally fast or good, and the percentage is essentially arbitrary),then one of the channels may be selected for single transmission basedon the quality of service being acceptable (e.g., exceeding athreshold). Said another way, if the quality of service of the primarychannel is above a threshold, it may be used in single channeltransmission rather than dual channel transmission. Similar remarksapply if the secondary channel exceeds the quality of service threshold.

In addition, note that there may generally be a preference for using oneof the RATs over the other. For example, since WiFi data is typicallynot metered or charged, but cellular data is, various embodiments mayprefer using WiFi over cellular whenever possible, e.g., until athreshold quality level cannot be maintained. As one example, it may bepreferable to establish WiFi to WiFi channels as primary channelswhenever the quality is acceptable (e.g., as opposed to using cellularfor any of the legs of the channels). Similarly, secondary channels mayprefer to use at least one WiFi leg of the channel, if possible.However, if the primary channel is WiFi to WiFi, it may be preferablefor the secondary channel to be cellular to cellular (or, moregenerally, that the primary and secondary channels do not overlap inindividual links) in order to ensure that the real-time application canbe maintained, even when a link (e.g., one side's WiFi or cellular)fails completely. This preference may also extend to disabling thesecondary channel, e.g., so that, if at all possible, the redundanttransmission is disabled as soon as possible, while maintaining adesirable quality for the real-time application, in order to reducecharges to the user, e.g., by disabling the channel using cellularwhenever possible.

Further, while examples above use cellular (e.g., which may be LTE) andWiFi, other embodiments are envisioned using any two or more RATs, asdesired (e.g., multiple cellular RATs).

FIG. 5—Using a Primary and Secondary Channel for a Real-Time Application

FIG. 5 is a flowchart diagram illustrating a method for using a primaryand secondary channel for a real-time application. The method may beperformed by a mobile device (such as mobile device 106) and a remotedevice. In the flowchart of FIG. 5, the mobile device 106 performs stepsof 502, 504, 506, 518 and 522. The first and second channels may use twodifferent RATs (e.g., WiFi and LTE, although other combinations areenvisioned). The method shown in FIG. 5 may be used in conjunction withany of the systems or devices shown in the above Figures, among otherdevices. In various embodiments, some of the method elements shown maybe performed concurrently, in a different order than shown, or may beomitted. Note also that additional method elements may also be performedas desired. The method may be performed as follows.

In 502, the mobile device 106 communicates packets in a-real timeapplication with a remote device using a single primary channel. Asdiscussed above, the real-time application may involve two waycommunication between the mobile device and a remote device, such asconferencing (e.g., videoconferencing), or may be largelyunidirectional, such as streaming information to the mobile device,although feedback from the mobile device may still occur, but maygenerally involve substantially less feedback communication than thestreaming information.

In general, the real-time application may involve audio and/or videopackets being communicated either bidirectionally or unidirectionally,depending on the nature of the real-time application. Initially, themobile device and remote device may communicate only using a primarycommunication channel. For example, the mobile device may use a firstRAT, such as a WiFi RAT, for the primary communication channel.

While performing the real-time application, in 504 the mobile device 106may monitor incoming packets to ensure that incoming packets arecontinually being received. Stated another way, the mobile devicemonitors incoming packets to obtain “No-Remote” information.

In 506 the mobile device determines if no packets have been receivedfrom the remote device for a first threshold of time. For example, thefirst threshold of time may be approximately three seconds, althoughother variations are envisioned. For example, the first threshold oftime could be smaller (e.g., 2 seconds, 1 second, 500 milliseconds, 100milliseconds, etc.) or larger (e.g., 4 seconds, 5 seconds, etc.) asdesired. The determination in 506 may be used to determine the downlinkchannel quality of the mobile device and/or the uplink channel qualityof the remote device.

If no packets have been received in the first threshold of time, themethod may advance to 522. However, if packets have been received by themobile device in the first threshold of time, the method may return to504 and the mobile device may continue the monitoring of incomingpackets. The mobile device may perform its determination periodically,such as every 5 seconds, 2 seconds, 1 second, 500 milliseconds, etc., asdesired. Alternatively, the mobile device may perform this determinationcontinuously.

In 512 the remote device may monitor its incoming packets as well todetermine if no packets have been received by the remote device in asecond threshold of time (i.e., to determine whether the remote devicehas received packets from the mobile device in the second threshold oftime). When the remote device determines that packets have been receivedwithin the second threshold of time, operation returns to 512, and theremote device continues monitoring its received packets. The remotedevice may perform its determination periodically, such as every 5seconds, 2 seconds, 1 second, 500 milliseconds, etc., as desired.Alternatively, the remote device may perform this determinationcontinuously.

When the remote device determines that no packets have been received in514, then in 516 the remote device sends feedback to the mobile device106. This feedback indicates that no packets have been received by theremote device in the second threshold of time. This feedback informationmay be referred to as “remote-no-remote” information.

In 518, the mobile device receives the feedback from the remote deviceand hence determines that no packets have been received by the remotedevice in a second threshold of time. In response to receiving thefeedback information from the remote device, the method advances to 522.

In 522 the mobile device begins using both the primary and secondarychannel for communication of packets in the real-time application. Thusif either the mobile device 106 detects non-receipt of incoming packetsfor the first threshold of time, or the remote device detectsnon-receipt of incoming packets for the second threshold of time andalerts the mobile device, then the mobile device 106 begins using asecondary channel alongside the primary channel. Thus in 522, based ondetermining that no packets have been received 1) by the mobile devicewithin the first threshold of time and/or 2) by the remote device withinthe second threshold of time, the mobile device and/or remote device maybegin using two channels (the primary communication channel and asecondary communication channel) for communicating the packets of thereal-time application. In various embodiments, the secondarycommunication channel may have already been established (e.g., duringinitial negotiation of channel set-up for the real-time application,such as approximately at the same time or soon after establishing theprimary communication channel) or may be established in 522 in responseto 506 and/or 514-518. If established in 522, the nature of thesecondary channel may have already been negotiated or may be determinedduring the establishment, as desired.

The first and second thresholds may be the same or different, asdesired. In one embodiment, the first threshold may be longer than thesecond threshold. For example, the second threshold may be approximatelytwo seconds, although smaller (e.g., 1 second, 500 milliseconds, 100milliseconds, etc.) or larger (e.g., 3 seconds, 4 seconds, etc.) valuesare also envisioned.

The mobile device may determine if no packets have been received by theremote device in a variety of manners. For example, as described abovethe remote device may send a message (feedback information) to themobile device indicating that no packets have been received within thesecond threshold of time (e.g., using the primary communication channeland/or another channel, such as a signaling channel). As anotherpossibility, the remote device may periodically send a messageindicating the period of time since a last packet has been received andthe mobile device may use that message to compare the reported time tothe second threshold of time. Other methods of determining if no packetshave been received by the remote device in the second threshold of timeare also envisioned. Note that the mobile device may similarly transmitthis information to the remote device (i.e., the two devices mayimplement symmetrical methods), as desired.

In one embodiment, using two channels for the real-time application maymean that substantially all packets (e.g., all audio and/or videopackets) of the real-time application are transmitted over both theprimary communication channel and the secondary communication channel.For example, in a videoconferencing application, the mobile device maytransmit the same packets (e.g., comprising audio and/or videoinformation of the videoconference) over both the primary communicationchannel and the secondary communication channel. As another possibility,the mobile device may encode the packets differently for each channel(e.g., where the audio and/or video packets are encoded according to adifferent codec, bit rate, and/or resolution), but the same audio andvideo information (albeit in different formats) is transmitted over bothchannels. Thus, the two channels may be used for redundant transmissionin either an identical sense (i.e., the same packets are sent) or in acontent sense (i.e., the same content is sent, but potentially in adifferent format). Thus, in either embodiment, in 508, the packets ofthe real-time application may be transmitted concurrently over the twochannels, in a redundant fashion. Accordingly, in some embodiments, thesecondary channel may be referred to as a redundant channel.

The method of FIG. 5 may also be extended to apply to more than twochannels, as desired.

FIG. 6—Transitioning from Dual Channel to Single Channel Transmission

FIG. 6 is a flowchart diagram illustrating a method for transitioningfrom using a primary and secondary channel for a real-time applicationto using a single channel for the real-time application. The method maybe performed by a mobile device (such as mobile device 106) and thefirst and second channels may use two different RATs (e.g., WiFi andLTE, although other combinations are envisioned). The method shown inFIG. 6 may be used in conjunction with any of the systems or devicesshown in the above Figures, among other devices. For example, the methodof FIG. 6 may be used in conjunction with the method of FIG. 5, e.g.,for transitioning between dual channel and single channel transmissions.However, the methods of FIGS. 5 and 6 may also be used independently(e.g., where a different method is used for entering or exiting singlechannel or dual channel transmission modes). In various embodiments,some of the method elements shown may be performed concurrently, in adifferent order than shown, or may be omitted. Note also that additionalmethod elements may also be performed as desired. The method may beperformed as follows.

In 602, a real-time application may be performed or executed, similar to502 above. However, in 602, the real-time application may use both aprimary and a secondary channel for communicating the packets of thereal-time application, e.g., in the manner described in 508 above.

In 604, a percentage of used packets for one or both of the primary andsecondary channels may be determined. In particular, in 602, the mobiledevice may transmit and receive duplicates of the same packets (or samecontent, but transmitted according to a different format) of thereal-time application using both the primary and secondary channels. Onthe receiving end, for a single packet, the mobile device may usewhichever of the duplicate packets is successfully received first, fromeither channel. Thus, if the packet is successfully received over theprimary channel before the secondary channel, the packet of the primarychannel may be used (e.g., placed in the receiving buffer, e.g., theaudio or video buffer) and the packet of the secondary channel may bediscarded. When packets are not successfully received (e.g., they arelost due to poor channel quality), then they are clearly not used. Basedon this procedure, the mobile device may maintain statistics for thepercentage of packets that are used for each channel. For example, themobile device may determine that 80% of the used packets come from theprimary channel versus 20% from the secondary channel, e.g., based onkeeping track of the packets on a packet-by-packet basis.

In 606, the determined percentage of 604 may be compared to a threshold.Additionally, in 608, if the determined percentage passes the threshold,the mobile device and remote device may transition from dual channelcommunication to single channel communication. For example, the methodmay transition from using both channels to using whichever channelexceeds the threshold. As a specific example, if the threshold is 75%and 80% of the packets from the primary channel are used, the method maytransition to using the primary channel and cease using the secondarychannel for communication. Similarly, if the threshold is 95% and 98% ofthe secondary channel packets are used, then the method may transitionto only using the secondary channel (which may be upgraded to theprimary channel) and may cease using the previous primary channel (whichmay now be the secondary channel or may be discarded entirely, e.g., ifthe channel has failed).

As another possibility, the threshold may be a lower threshold. Forexample, the method may disable use of a channel if the packets usedform that channel fall below a threshold percentage, such as 5%, 10%,20%, 25%, etc. Thus, if the channel's use rate is too low to be useful(e.g., most of the packets are not arriving, or the other channel issimply much faster or better), then that channel may be disabled, andthe other channel may be used. For example, if only 6% of the packetsfrom the secondary channel are used and the threshold is 10%, then thesecondary channel may be disabled and the method may simply use theprimary channel for communication, in a single channel mode.

While the steps above have been described from the mobile device's pointof view, the same method may be performed by the remote device. Inaddition, the two devices may compare the results of their comparisonsto thresholds and determine whether to transition to a single channelmode based on the results. For example, the method may only transitionfrom dual channel transmission to single channel transmission if bothdevices agree that the transition should occur and also agree on theparticular channel. Alternatively, the transition may occur if one ofthe devices determines that the transition should occur, and the otherdevice does not have a reason to object (e.g., the other device mayobject if that channel is not sufficient, based on thresholds from itsend). As an example, if the remote device indicates that the comparisonindicates that the primary channel is sufficient and that dual channeltransmission is no longer necessary, but the mobile device determinesthat it is only using 2% of the packets form the primary channel, it mayoverride the decision from the remote device.

As discussed above, while the method of FIG. 6 may result from thetransition from FIG. 5, other methods of transitioning from singlechannel transmission mode to dual channel transmission mode may be used,e.g., using different methods for determining channel degradation thanthose discussed in FIG. 5, such as packet loss ratios, round trip times,quality of service metrics, etc. Similarly, other methods than FIG. 6may be used to transition from dual channel transmission mode to singlechannel transition mode, e.g., in combination with the method of FIG. 5,such as using quality of service metrics, packet loss ratios, round triptimes, etc. Additionally, these other metrics may also be incorporatedinto the methods of FIGS. 5 and 6 as desired.

Finally, the method of FIG. 5 may also be extended to apply to more thantwo channels, as desired.

Modifying Settings Based on Changes in Channel Use

Video pipelines for real-time applications are usually optimized foruser scenarios. For instance, a video conferencing application on amobile device can have completely different camera capture settings,video codec, and network adaptation algorithms between differentchannels or RATs (e.g., between WiFi and cellular channels).Accordingly, when a video conferencing application switches transmissionbetween two different connections (channels), embodiments may operate todynamically select the video pipeline which is best suited for thisconnection type. The transition between different settings, such ascamera settings, pipeline settings, etc., may be performed dynamicallyduring operation and hence maybe be seamless without sacrificing userexperience.

In some embodiments, network adaptation algorithms may be changed duringthe transition. As an example, when encoder setting switches, packets inthe lower level queue are no longer useful, and may be flushed (e.g.,from the transmission queue or buffer) which may save data usage for thesender and/or receiver. Additionally, during the transition period,incoming refresh frame requests may be ignored until the new I-frame isreceived by the receiver. As another example, when sending frames overboth the primary and secondary channels, the amount of packets sent onthe redundant channel, e.g., the cellular channel, may be reduced, e.g.,to avoid data use or because the cellular channel cannot handle the datathroughput as compared to the WiFi connection. For example, droppableframes, such as certain P-frames and/or B-frames may not be sent over acellular channel, but may be sent over WiFi. In one embodiment,adaptation on different channels may be separate and each channel mayuse its own set of parameters.

FIG. 7—Modifying Settings Based on Changes in Channel(s) forTransmission

FIG. 7 is a flowchart diagram illustrating a method for modifyingsettings based on changes in channel transmission. The method formodifying settings based on channel transmission changes may beperformed dynamically or “on-the-fly” during operation. Thus, as oneexample, numerous such settings modifications may be performed during asession of a real-time application, as conditions warrant.

The method may be performed by a mobile device (such as mobile device106) and may involve the use of first and second channels that may usetwo different RATs (e.g., WiFi and LTE, although other combinations areenvisioned). The method shown in FIG. 7 may be used in conjunction withany of the systems or devices shown in the above Figures, among otherdevices. For example, the method of FIG. 7 may be used in conjunctionwith the methods of FIGS. 5 and 6. In various embodiments, some of themethod elements shown may be performed concurrently, in a differentorder than shown, or may be omitted. Note also that additional methodelements may also be performed as desired. The method may be performedas follows.

In 702, first settings may be used based on use of a first channel forcommunicating packets in a real-time application. The settings may beany of various ones related to capturing and transmitting the data forthe real-time application. For example, in one embodiment, the real-timeapplication may be a videoconference or more generally related to videodata and the settings may relate to the format, bit rate, resolution,etc. of the video packets. As a specific example, for a videoconference,the settings may include camera capture settings (e.g., includingresolution) used by the mobile device to capture images used for thevideoconference. The settings may also relate to a type of video encoderused in encoding captured video for transmission during the real-timeapplication.

In particular, in one embodiment, if the first channel uses a highbandwidth RAT and/or does not charge for increased data usage, higherquality (e.g., higher resolution, bit rate, etc.) settings may be used.For example, if the mobile device uses WiFi for the first channel, thenhigh quality settings may be used. Alternatively, if the mobile deviceuses a RAT that cannot support the high quality settings and/or one thatcharges for increased data use (or where data use is generally desiredto be kept low), lower quality (e.g., lower resolution, bit rate, etc.)settings may be used. For example, if the mobile device uses cellularcommunication for the first channel, then low quality settings may beused.

In some embodiments, the settings may be based only on the local link(e.g., the link from the mobile device to the network, such as WiFi orcellular) and may ignore the remote link. For example, high qualitysettings may be used if WiFi is used by the mobile device, ignoringwhether the remote device uses WiFi or cellular. Alternatively, thesettings may be based on both the local link and the remote link (e.g.,based on the characteristics of the end-to-end channel). For example, ifeither device uses cellular, then the mobile device may use the lowquality settings, e.g., even when the mobile device's local link isWiFi.

In 704, a second channel may be used for communicating the packets. Insome embodiments, the second channel may be used concurrently with thefirst channel (e.g., in the redundant fashion discussed previously) ormay be used instead of the first channel, depending on the embodiment.Note that the second channel may be used in response to initiation bythe mobile device and/or by initiation by the remote device, as desired.Thus, the mobile device may cause the use of the second channel, or maydetermine that the second channel is being used by the remote device.

In 706, in response to using the second channel, second settings may beused. Similar to descriptions above, the settings for the second channelmay be based on the local link (e.g., WiFi or cellular) or may be basedon the overall link (e.g., if the remote device's local link haschanged, even though the mobile device's has not).

Where the second channel is used as the only channel (i.e., notconcurrently with the first channel), then the choice of settings may besimilar to that described above for the first settings for the firstchannel. More specifically, in one embodiment, if the second channeluses a high bandwidth RAT and/or does not charge for increased datausage, higher quality (e.g., higher resolution, bit rate, etc.) settingsmay be used. For example, if the mobile device uses WiFi for the secondchannel, then high quality settings may be used. Alternatively, if themobile device uses a RAT that cannot support the high quality settingsand/or one that charges for increased data use (or where data use isgenerally desired to be kept low), lower quality (e.g., lowerresolution, bit rate, etc.) settings may be used. For example, if themobile device uses cellular communication for the second channel, thenlow quality settings may be used.

However, where the first and second channels are used concurrently,e.g., in a redundant fashion, the choice of settings may be affected.For example, in one embodiment, the same settings may be used for boththe first and second channels (e.g., where the first and second channelsshare a same encoder). In this case, the choice of settings may need tobe determined based on both channels, rather than only based on thesecond channel. For example, the single settings used for both may bebased on the channel that has the lowest bandwidth or other constraints.As a specific example, if the single setting is based only on the mobiledevice's local links for the first and second channel (and thereforeignores the links of the remote device), the lowest settings associatedwith either the first or second channel may be used. For example, if themobile device uses WiFi for the first channel (e.g., which normallywould use a high quality setting) and cellular for the second channel(e.g., which normally would use a low quality setting), then the lowquality setting may be used as the single setting (e.g., since thebandwidth of the cellular connection may not be supported, the data useon the cellular connection may be too high, etc.).

Similarly, if the single setting is based on the end-to-endcharacteristics of the channel, and therefore may be affected by thelocal link of the remote device, then the lowest quality setting may beused from all of the links. For example, if the mobile device uses WiFifor both the first and second channels, and the remote device uses WiFifor the first channel and cellular for the second channel, then thesettings may be set based on the settings that would be used for thecellular link of the second channel of the remote device, e.g., sincethis setting would be the lowest of the others, e.g., based on bandwidthor data usage. Note that while the lowest setting is considered here, itis also possible that an average setting or highest setting may be used,in other embodiments.

When the setting is changed based on using the second channel, and wherea single encoder is used, the encoder (e.g., where it is implemented insoftware) may be torn down and rebuilt for the second settings (e.g.,which may be a different type of encoder, different bit rate, differentformat, different resolution, etc.). Additionally, the capture settingsof the camera may be modified to reflect the changes, e.g., inresolution.

Where the first and second channels are used concurrently, but thesetting is not shared across the first and second channels (e.g., whereseparate encoders are used for each channel), then the setting may beselected independently for each channel, e.g., similar to discussionsabove regarding the selection method for the first channel. When twodifferent settings are used, the mobile device may implement twodifferent encoders (e.g., software encoders) may be used, e.g., a highquality encoder and a low quality encoder, and each encoder may beassociated with a respective channel of the first and second channels.

FIG. 8—Dynamic Modification of Camera Capture Settings

FIG. 8 is a flowchart diagram illustrating dynamic modification ofcamera capture settings in a mobile device based on changes in thecommunication channel(s) being used.

As shown, in 802 the mobile device is configured with first cameracapture settings for use with a real-time application. The first cameracapture settings may be based on the particular communication channelbeing used. For example, if the mobile device is using a high qualityconnection, such as a WiFi connection, the first camera capture settingsmay specify a higher resolution for acquired images. If the mobiledevice is using a lower quality connection, such as a cellularconnection, the first camera capture settings may specify a lowerresolution for acquired images.

In 804 the mobile device may begin using a second channel forcommunicating the packets. In some embodiments, the second channel maybe used concurrently with the first channel (e.g., in the redundantfashion discussed previously) or may be used instead of the firstchannel, depending on the embodiment. Note that the second channel maybe used in response to initiation by the mobile device and/or byinitiation by the remote device, as desired. Thus, the mobile device maycause the use of the second channel, or may determine that the secondchannel is being used by the remote device.

In 806 the mobile device may dynamically configure its camera withsecond camera capture settings that replace the first camera capturesettings. The mobile device dynamically configures its camera withsecond camera capture settings based on the change in using the secondcommunication channel determined in 804. The second camera capturesettings may have a different resolution for acquiring captured imagesthan the resolution of the first camera capture settings. The variousoperations described in 706 of FIG. 7 may be performed here.

Thus, in one embodiment, camera capture settings may be updated based onlocal and/or remote party information. For example, if a local mobiledevice switches from transmitting on WiFi to transmitting throughcellular, the resolution of the mobile device camera may be modifiedfrom a higher resolution to a lower resolution. When the mobile deviceswitches from cellular to WiFi, the resolution of the mobile devicecamera may also be switched from the lower resolution to the higherresolution. Similarly, if the remote device switches from WiFi tocellular, the resolution of the local mobile device camera may beswitched from the higher resolution to the lower resolution.Additionally, if the remote device switches from cellular to WiFi, theresolution of the local mobile device camera may be switched form thelower resolution to the higher resolution. The switching betweendifferent camera resolutions may occur dynamically or “on-the-fly”during operation of the mobile device in response to changingtransmission characteristics of the mobile device as described above.

In some embodiments, the switching between resolution may be symmetric(where both the mobile device and the remote device change resolutionsto higher or lower resolutions) or asymmetric (where each individualdevice elects to change its resolution independently). For example, forsymmetric modifications, if both devices are on WiFi, then each devicemay use a higher resolution (either the same or different higherresolution, as desired). Then, if one device switches from WiFi tocellular, both devices may use a lower resolution (either the same ordifferent lower resolution, as desired). For asymmetric modifications,on the other hand, if one device switches from WiFi to cellular, thanthat device may switch to the lower resolution, while the other devicemay maintain the higher resolution, since it is still using WiFi, as oneexample.

FIG. 9—Dynamic Modification of Camera Capture Settings and Video Encoder

FIG. 9 is a flowchart diagram illustrating dynamic modification ofcamera capture settings and video encoder usage in a mobile device basedon changes in the communication channel(s) being used.

As shown, in 902 the mobile device is configured with first cameracapture settings and with a first video encoder for use with a real-timeapplication. The first camera capture settings and first video encodermay be based on the particular communication channel being used. Forexample, if the mobile device is using a high quality connection, suchas a WiFi connection, the first camera capture settings may specify ahigher resolution for acquired images and the first video encoder mayperform encoding presuming the presence of a high quality channel. Ifthe mobile device is using a lower quality connection, such as acellular connection, the first camera capture settings may specify alower resolution for acquired images, and the first video encoder mayperform encoding presuming the presence of a lower quality channel.

In 904 the mobile device may begin using a second channel forcommunicating the packets. In some embodiments, the second channel maybe used concurrently with the first channel (e.g., in the redundantfashion discussed previously) or may be used instead of the firstchannel, depending on the embodiment. Note that the second channel maybe used in response to initiation by the mobile device and/or byinitiation by the remote device, as desired. Thus, the mobile device maycause the use of the second channel, or may determine that the secondchannel is being used by the remote device.

In 906 the mobile device may dynamically configure its camera withsecond camera capture settings that replace the first camera capturesettings. The mobile device dynamically configures its camera withsecond camera capture settings based on the change in using the secondcommunication channel determined in 904. The second camera capturesettings may have a different resolution for acquiring captured imagesthan the resolution of the first camera capture settings. Any of theoperations described in 706 of FIG. 7 may be performed here.

In 908 the mobile device dynamically tears down the first video encoderand creates a new second video encoder with different settings. The newsecond video encoder is designed to encode images

Thus, in one embodiment, both the camera capture settings and the videoencoder in use may be updated based on local and/or remote partyinformation. For example, if a local mobile device switches fromtransmitting on WiFi to transmitting through cellular, the resolution ofthe mobile device camera may be modified from a higher resolution to alower resolution, and also a new video encoder may be created for usewith this lower resolution. When the mobile device switches fromcellular to WiFi, the resolution of the mobile device camera may also beswitched from the lower resolution to the higher resolution, and thevideo encoder changed accordingly. Similarly, if the remote deviceswitches from WiFi to cellular, the resolution of the local mobiledevice camera may be switched from the higher resolution to the lowerresolution, and the video encoder changed accordingly. Additionally, ifthe remote device switches from cellular to WiFi, the resolution of thelocal mobile device camera may be switched form the lower resolution tothe higher resolution, and the video encoder changed accordingly. Theswitching between different camera resolutions and/or different videoencoders may occur dynamically or “on-the-fly” during operation of themobile device in response to changing transmission characteristics ofthe mobile device as described above.

In each of the methods described herein, e.g., in each of FIGS. 7-9, themobile device and/or the remote device may be configured to dynamicallyreact to the change in settings of the other device. For example, whenthe mobile device modifies its settings based on using the secondchannel, the remote device may dynamically react to the change insettings in various different manners.

FIG. 10—Use of I Frames to Notify the Remote Device of Changed Settings

In one embodiment, the mobile device may send a new I-frame to theremote device upon the change in settings, and the I-frame may includemeta-data identifying the characteristics of the new setting being used(e.g., the resolution, encoder type, bit rate, etc.). The remote devicemay receive this I-frame and create or modify a decoder in order toadjust to the new settings.

In some cases, the I-frame may be missed by the remote device, and themobile device may retransmit the I-frame (or a new I-frame) in response,e.g., based on receiving a message requesting an I-frame from the remotedevice. For example, as shown in FIG. 10 at 942 a newly created videoencoder in the mobile device may transmit an I-frame to the remotedevice to notify the remote device of the new encoder. However, in someinstances, this notification I-frame may get lost in the network orotherwise is not received by the remote device. For example, at 944 themobile device determines if an acknowledgement has been received forthis I frame. If an acknowledgement has been received for this I frame,then operation completes. If the mobile device has not received anacknowledgement for this “notification” I frame within a certain timeperiod, the new encoder operates to retransmit this notification I-frameinstead of a regular refresh frame, such as a P or B frame. In thisinstance, if the I frame was not received by the receiver then thereceiver will not be informed that the encoder settings have changed.Thus a retransmission of this notification I-frame is warranted.

As another possibility, when a new video setting is selected, a newI-frame may be generated by the video encoder with different codingparameters, such as picture width, height, etc. When the receiverreceives this new I-frame, it may detect this setting change and handleit correctly. However, when this new I-frame is lost or not received bythe receiver, the receiver may receive later P-frames coded by the newencoder. Because the receiver is still using the old settings, it willnot be able to decode these new P-frames. Accordingly, an decoder errormay be generated, and the receiver may send out a message, e.g., arequest to the sender asking for a new “Refresh” frame using a request,e.g., a Full Intra-request (FIR) in RTCP. Once the sender receives therequest, it may typically send out Intra-P frame in normalimplementation (e.g., because an Intra-P frame is normally smaller thana new I-frame). However, in the case of a change in settings, the sendermay determine that the coding parameters of the request and determine ifthey match the current settings. If the parameters do match, then thesender may generate and transmit an Intra-P frame in response to therequest. However, if the parameters do not match, then the sender maygenerate and transmit a new I-frame (with the new settings) to thereceiver. This embodiment may not require a change to existing receiverbehavior and may therefore be back-ward compatible.

Additionally, or alternatively, the mobile device may send a separatemessage identifying the change in settings. As a further possibility,the two devices may independently be aware of the use of the secondchannel and may both choose a same new setting based on the use of thesecond channel (e.g., based on pre-stored or pre-arranged associationsof settings with channels or channel types). Accordingly, both devicesmay already know what the new setting will be and no substantivecommunication of that new setting may be necessary. Note that whilethese descriptions are discussed from the point of view of the mobiledevice to the remote device, the mobile device may also implement thesame functionality for changes in settings from the remote device to themobile device.

Therefore, in summary, in switching between resolutions, camerasettings, bit rates, resolution, codecs, the mobile device may use asingle video codec instance to handle the change. For example, themobile device may execute a single video encoder, and when a camerasetting change occurs, it may tear down the encoder and recreate a newone with different settings. The mobile device may send an I frameencoded with metadata (“notification” I frame) to notify the receiverdecoder of the new video encoder in the mobile device. Thus, when asender encoder generates a new “notification” I-frame and transmits it,when the receiver receives the new I-frame, the receiving device maychange its decoder to the new settings. However, when the I-frame fromthe newly created encoder is lost or otherwise not received by thereceiver device, the receiver will not be aware that the encoder haschanged, and hence its decoder will not be adjusted accordingly. Whenthis happens, the sender may be configured to retransmit the I-frameinstead of a regular refresh frame, e.g., during a settings changeoveror in response to a request from the receiver.

When a mobile device uses a single encoder on conjunction with multiplecommunication channels, as discussed in the previous Figures, the sendermay send the same frames or packets (i.e., with the same resolutions orformats) across both channels. Accordingly, either packet from eitherchannel can be provided to the receiver's decoder interchangeably. Thuspackets provided by the single encoder can be provided on to eitherchannel, and can be fed to the decoder in an interchangeable fashion. Inother words, some packets received on the primary communication channelcan be fed to the receiver decoder, followed by packets received on thesecondary communication channel being fed to the receiver decoder, andso on.

However, while using a single encoder instance (which is dynamicallytorn down and a new one created in its place as needed) may be simple toimplement, it may have some limitations. Instead, multiple encoderinstances may be desirable, depending on the implementation (e.g., fortransition speed, efficiency, data use, etc.). In this embodiment, thesender may implement two encoders with different settings, which may beselected based on respective channels (e.g., WiFi or cellular). The twoencoders may be the same codec (e.g., H.264) but with different settingsor may be two different codecs (e.g., H.264 and HEVC), as desired. Themultiple video codec instances may be based on the respectivecommunication channel, e.g., a first video codec instance may betailored to the primary channel and a second video codec instance may betailored to the secondary channel. For example, the mobile device mayhave two different H.264 video encoder instances with different videosettings for reach respective channel. The sending device may selectwhich of the video encoder instances to used based on the currentconnection that is being used.

When using multiple channels, as discussed in the previous Figures,using multiple encoders may mean that packets or frames from the sendercannot be used interchangeably, resulting in less redundancy and moreoverhead. However, it is also possible that the decoder(s) could beconfigured to use scalable video coding (SVC) or generally use packetsor frames from each channel, e.g., where that packet or frame is notavailable on one of the channels. For example, the various frames may bedecoded by either a single decoder or two different decoders and thencombined for video display after decoding.

Scalable video coding (SVC) may require configuration of both theencoder (on the transmitting device) and the decoder (on the receivingdevice) for proper operation. It is noted that SVC may not require twodecoders. For example, a single SVC decoder can decode multiple layersof the SVC bitstream. An example of using SVC in this case is that datais concurrently transmitted on both channels, the video can be codedusing two layers (a base layer and an enhancement layer). Theenhancement layer is used to encode the additional video resolution,frame rate or quality on top of the base layer. Then the base layer maybe transmitted on only one channel (e.g., the cellular channel), andboth the base layer and the enhancement layer may be transmitted on theother channel (WiFi). When the decoder receives base layer packets fromeither channel, it can reconstruct the video quality of the base layer.When the decoder also receives all or part of the enhancement layer, itcan reconstruct the higher quality frames.

FIG. 11 illustrates an exemplary embodiment where different amounts ofdata are sent over each channel. In particular, the base layer andenhancement layer of the video may be transmitted from the firstwireless device 106 to the second wireless device 154 over the primarychannel in 1102. Concurrently, the same base layer may be transmittedover the secondary channel in 1104. The second wireless device 154 maygenerate the base layer using the transmissions of 1102 and 1104. Forexample, where a base layer packet was successfully transmitted via theprimary channel in 1102, but was not successfully transmitted via thesecondary channel in 1104, the second wireless devices 154 may use thesuccessfully transmitted base layer packet from the primary channel.Similarly, where a base layer packet was not successfully transmittedvia the primary channel in 1102, but was successfully transmitted viathe secondary channel in 1104, the second wireless device 154 may usethe base layer packet from the secondary channel. Thus, the base layerpackets may be redundantly transmitted over both channels and the secondwireless device may generate the base layer using successfully receivedpackets from either channel, as desired. In 1108, the second wirelessdevice 154 may combine or enhance the base layer with availableenhancement layer packets. For example, where both a base layer frameand a corresponding enhancement layer frame are successfully received,the second wireless device 154 may enhance the base layer frame with theenhancement information to generate a higher quality frame (e.g., higherresolution0 than would be available by simply using the base layerframe. Alternatively, or additionally, the enhancement layer frame mayprovide more intermediary frames or intermediary information to providea higher quality video experience. The enhancement layer information maybe used opportunistically, where the enhancement information issuccessfully received in 1102. Accordingly, the resulting video may bedisplayed in 1110.

Therefore, when sending packets over both channels, the amount ofpackets or data sent over one of the channels (e.g., the cellularchannel or redundant channel) may be reduced. Note that while both thebase layer and the enhancement layer are shown as sent over the primarychannel, it is possible that they might be sent over the secondarychannel and the primary channel may only send the base layer. Forexample, if the primary channel includes a cellular connection and thesecondary channel does not include a cellular connection, the primarychannel may not send the enhancement layer to avoid using too much ofthe user's data, while the secondary channel (e.g., which may use WiFi),may transmit both the base layer and the enhancement layer.

In one embodiment, frames may be partitioned into “non-droppable”frames, such as I frames and possibly certain P frames, and droppablesuch as B-frames and certain (or all) P-frames. Both non-droppable anddroppable frames may be sent over one channel (e.g., the primary channelor WiFi channel). Thus all of the I, P and B frames may be sent over theprimary channel. The secondary channel (e.g., the redundant channel orcellular channel) may transmit only the non-droppable frames only anddoes not transmit the droppable frames. In other words, the secondarychannel may transmit only the I frames and perhaps certain P frames, anddoes not transmit frames that can be dropped without seriously impactingimage quality. In some embodiments, the adaptation or manner thatinformation is sent may be independent for each channel, e.g., eachhaving its own set of parameters, as desired.

In one embodiment, a single encoding session is used wherein the encodergenerates one bitstream with a portion of the bitstream comprisingnon-droppable frames and a second portion of the bitstream comprisingdroppable frames. The mobile device can then decide on how many of thedroppable frames are transmitted on each channel. For example, themobile device can transmit all of the non-droppable frames and a largepercentage, or all, of the droppable frames on the primary (highquality) channel. The mobile device can then transmit all of thenon-droppable frames and a smaller percentage, or none, of the droppableframes on the secondary (lower quality) channel.

FIG. 12 illustrates an exemplary embodiment where droppable frames aresent over the primary channel and only the non-droppable frames are sentover the secondary channel, although portions of the droppable framesare also envisioned for the secondary channel. In the embodiment shown,the first wireless device 106 may send both droppable and non-droppableframes of a video over the primary channel in 1202. Concurrently, thefirst wireless device 106 may transmit only the non-droppable framesover the secondary channel in 1204. For example, all I, P, and B framesmay be transmitted in 1202; however, B frames (and/or possibly certain Pframes, as determined by the first wireless device 106) may not betransmitted in 1204, allowing for lower transmission of data over thesecondary channel. In 1206, similar to 1108 above, the second wirelessdevice 154 may opportunistically combine the droppable and non-droppableframes. For example, where an I frame is successfully transmitted overthe first channel in transmission 1202, but is not successfullytransmitted over the second channel in transmission 1204, the secondwireless device 154 may use the successfully received I frame from thetransmission 1202. Conversely, where an I frame is not successfullytransmitted over the first channel in transmission 1202, but issuccessfully transmitted over the second channel in transmission 1204,the second wireless device 154 may use the successfully received I framefrom the transmission 1202. Since the droppable frames are only sentover the primary channel, in this case, the second wireless device 154may insert the droppable frames at their appropriate positions in thevideo stream when they are successfully received. After combining thesedroppable and non-droppable frames, the video from the combination maybe displayed in 1208.

Therefore, when sending packets over both channels, the amount ofpackets or data sent over one of the channels (e.g., the cellularchannel or redundant channel) may be reduced. Note that while both thedroppable and non-droppable frames are shown as sent over the primarychannel, it is possible that they might be sent over the secondarychannel and the primary channel may only send the non-droppable frames.For example, if the primary channel includes a cellular connection andthe secondary channel does not include a cellular connection, theprimary channel may not send the droppable frames to avoid using toomuch of the user's data, while the secondary channel (e.g., which mayuse WiFi), may transmit both the droppable and non-droppable frames.

During the transition period for the changing settings, various networkadaptation algorithms may be used. For example, during a change ofsettings, data associated with the prior settings may no longer benecessary. Specifically, buffered data for the previous settings may bediscarded (e.g., the encoded data to be transmitted, which may save datausage, and/or the received encoded data, if the decoder has already beenreplaced). Additionally, incoming requests (e.g., from the receivingdevice) to provide new information (e.g., incoming refresh framerequests) may be ignored until the settings have been changed (e.g., thenew I-frame of the new setting has been received by the receivingdevice), since such data will soon be out of date.

Specific Embodiments

The following numbered paragraphs describe specific embodiments that donot limit the descriptions provided herein.

1. A method for performing a real-time application on a mobile device,wherein the real-time application communicates audio/video packets witha remote device, the method comprising, at the mobile device:communicating the audio/video packets on a first communication channelwith the remote device; determining if no packets have been received bythe mobile device from the remote device for a first threshold period oftime, wherein said determining measures downlink channel quality of themobile device; if no packets have been received by the mobile devicefrom the remote device for the first threshold period of time,establishing a second communication channel for transmission of theaudio/video packets with the remote device.

2. The method of any of the preceding paragraphs, further comprising:determining if a message is received from the remote device indicatingthat no packets have been received by the remote device from the mobiledevice for a second threshold period of time, wherein said determiningmeasures uplink channel quality of the mobile device; if a message isreceived from the remote device indicating that no packets have beenreceived by the remote device from the mobile device for the secondthreshold period of time, establishing the second communication channelfor transmission of the audio/video packets with the remote device.

3. The method of any of the preceding paragraphs, wherein the firstthreshold period of time is longer than the second threshold period oftime.

4. The method of any of the preceding paragraphs, further comprising:after said establishing the second communication channel, communicatingthe audio/video packets on the second communication channel concurrentlywith said communicating the audio/video packets on the firstcommunication channel.

5. The method of any of the preceding paragraphs, further comprising:measuring a percentage of audio/video packets received on the firstchannel that are used in audio/video playback by an audio/video decoderin the mobile device, wherein said measuring is performed during saidcommunicating the audio/video packets concurrently on the first andsecond communication channels; disabling transmission of the audio/videopackets over the second channel if the percentage of receivedaudio/video packets on the first channel that are used in audio/videoplayback is above a first threshold.

6. The method of any of the preceding paragraphs, further comprising: ifthe percentage of audio/video packets received on the first channel thatare used in audio/video playback is below a second threshold, performingtransmission of the audio/video packets exclusively on the secondchannel.

7. The method of any of the preceding paragraphs, further comprising:measuring a percentage of audio/video packets received on the secondchannel that are used in audio/video playback by an audio/video decoderin the mobile device, wherein said measuring is performed during saidcommunicating the audio/video packets concurrently on the first andsecond communication channels; disabling transmission of the audio/videopackets over the first channel if the percentage of received audio/videopackets on the second channel that are used in audio/video playback isabove a first threshold.

8. The method of any of the preceding paragraphs, further comprising:measuring a first percentage of audio/video packets received on thefirst channel and a second percentage of audio/video packets received onthe second channel that are used in audio/video playback by anaudio/video decoder in the mobile device, wherein said measuring isperformed during said communicating the audio/video packets concurrentlyon the first and second communication channels; disabling transmissionof the audio/video packets over one of the first channel or the secondchannel based on the first percentage and the second percentage.

9. The method of any of the preceding paragraphs, wherein the firstcommunication channel uses a Wireless LAN (local area network) radioaccess technology (RAT), and wherein the second communication channeluses a cellular RAT.

10. The method of any of the preceding paragraphs, wherein the firstchannel comprises a primary communication channel and wherein the secondchannel comprises a redundant communication channel.

11. A mobile device configured to perform a real-time application,wherein the real-time application communicates audio/video packets witha remote device, the mobile device comprising: at least one antenna; atleast one radio coupled to the at least one antenna, wherein the atleast one radio is configured to perform communication using a firstradio access technology (RAT) and a second RAT and maintain a connectionto both the first RAT and the second RAT concurrently; and one or moreprocessors coupled to the first radio, wherein the one or moreprocessors and the first radio are configured to: communicate theaudio/video packets on a first communication channel with the remotedevice; determine one or more of: 1) if no packets have been received bythe mobile device from the remote device for a first threshold period oftime, wherein said determining measures downlink channel quality of themobile device; or 2) if a message is received from the remote deviceindicating that no packets have been received by the remote device fromthe mobile device for a second threshold period of time, wherein saiddetermining measures uplink channel quality of the mobile device; ifeither 1) or 2) are determined to have occurred, establish a secondcommunication channel for transmission of the audio/video packets withthe remote device.

12. The mobile device of any of the preceding paragraphs, wherein thefirst threshold period of time is longer than the second thresholdperiod of time.

13. The mobile device of any of the preceding paragraphs, wherein theone or more processors and the first radio are further configured to:after establishment of the second communication channel, communicate theaudio/video packets on the second communication channel concurrentlywith communication of the audio/video packets on the first communicationchannel.

14. The mobile device of any of the preceding paragraphs, wherein theone or more processors and the first radio are further configured to:measure a percentage of audio/video packets received on the firstchannel that are used in audio/video playback by an audio/video decoderin the mobile device, wherein said measuring is performed during saidcommunicating the audio/video packets concurrently on the first andsecond communication channels; disable transmission of the audio/videopackets over the second channel if the percentage of receivedaudio/video packets on the first channel that are used in audio/videoplayback is above a first threshold.

15. The mobile device of any of the preceding paragraphs, wherein theone or more processors and the first radio are further configured to: ifthe percentage of audio/video packets received on the first channel thatare used in audio/video playback is below a second threshold, performtransmission of the audio/video packets exclusively on the secondchannel.

16. A non-transitory computer accessible memory medium comprisingprogram instructions for controlling operation of a mobile device,wherein the mobile device is configured to perform a real-timeapplication, wherein the real-time application communicates audio/videopackets with a remote device, wherein the program instructions areexecutable to implement: communicating the audio/video packets on afirst communication channel with the remote device; determining if nopackets have been received by the mobile device from the remote devicefor a first threshold period of time, wherein said determining measuresdownlink channel quality of the mobile device; if no packets have beenreceived by the mobile device from the remote device for the firstthreshold period of time, establishing a second communication channelfor transmission of the audio/video packets with the remote device.

17. The non-transitory computer accessible memory medium of any of thepreceding paragraphs, wherein the program instructions are furtherexecutable to implement: determining if a message is received from theremote device indicating that no packets have been received by theremote device from the mobile device for a second threshold period oftime, wherein said determining measures uplink channel quality of themobile device; if a message is received from the remote deviceindicating that no packets have been received by the remote device fromthe mobile device for the second threshold period of time, establishingthe second communication channel for transmission of the audio/videopackets with the remote device.

18. The non-transitory computer accessible memory medium of any of thepreceding paragraphs, wherein the program instructions are furtherexecutable to implement: after establishing the second communicationchannel, communicating the audio/video packets on the secondcommunication channel concurrently with said communicating theaudio/video packets on the first communication channel; measuring apercentage of audio/video packets received on the first channel that areused in audio/video playback by an audio/video decoder in the mobiledevice, wherein said measuring is performed during said communicatingthe audio/video packets concurrently on the first and secondcommunication channels; and disabling transmission of the audio/videopackets over the second channel if the percentage of receivedaudio/video packets on the first channel that are used in audio/videoplayback is above a first threshold.

19. The non-transitory computer accessible memory medium of any of thepreceding paragraphs, wherein the program instructions are furtherexecutable to implement: if the percentage of audio/video packetsreceived on the first channel that are used in audio/video playback isbelow a second threshold, performing transmission of the audio/videopackets exclusively on the second channel.

20. The non-transitory computer accessible memory medium of any of thepreceding paragraphs, wherein the first communication channel uses aWireless LAN (local area network) radio access technology (RAT), andwherein the second communication channel uses a cellular RAT.

1. A method for performing a real-time application on a mobile device,wherein the real-time application generates audio/video packets, themethod comprising: at the mobile device: communicating the audio/videopackets on a first communication channel; communicating the audio/videopackets on a second communication channel concurrently with saidcommunicating the audio/video packets on the first communicationchannel; measuring a percentage of received audio/video packets on thefirst channel that are used in audio/video playback by an audio/videodecoder in the mobile device, wherein said measuring is performed duringsaid communicating the audio/video packets concurrently on the first andsecond communication channels; disabling transmission of the audio/videopackets over the second channel if the percentage of receivedaudio/video packets that are used in audio/video playback is above afirst threshold.

2. The method of any of the preceding paragraphs, further comprising:performing transmission of the audio/video packets exclusively on thesecond channel, if the percentage of received audio/video packets thatare used in audio/video playback is below a second threshold.

3. The method of any of the preceding paragraphs, further comprising:determining degradation in the first communication channel; wherein saidcommunicating the audio/video packets on the second communicationchannel concurrently with the first communication channel is performedin response to determined degradation in the first communicationchannel.

4. The method of any of the preceding paragraphs, wherein saiddetermining degradation in the first communication channel comprises:measuring link quality for local uplink or remote downlink on the firstcommunication channel; and measuring link quality for local downlink orremote uplink on the first communication channel

5. The method of any of the preceding paragraphs, wherein the mobiledevice communicates with a remote device, the method further comprising:determining if no packets have been received by the mobile device fromthe remote device for a first threshold period of time, wherein saiddetermining measures downlink channel quality of the mobile device; ifno packets have been received by the mobile device from the remotedevice for the first threshold period of time, establishing the secondcommunication channel for transmission of the audio/video packets withthe remote device.

6. The method of any of the preceding paragraphs, wherein the firstcommunication channel is a primary communication channel; wherein thesecond communication channel is a redundant communication channel.

7. The method of any of the preceding paragraphs, wherein the firstcommunication channel is one of a WiFi communication channel or acellular communication channel; wherein the second communication channelis the other of the wireless LAN communication channel or the cellularcommunication channel.

1. A method for conducting a real time application on a mobile device,the method comprising: at the mobile device: capturing first images froma camera of the mobile device using first camera capture settings havinga first resolution; communicating first multimedia packets on a firstcommunication channel during a video session, wherein the firstmultimedia packets correspond to the first images captured from thecamera using the first camera capture settings having the firstresolution; in response to transitioning to communication of multimediapackets on a second communication channel during the video session,changing the camera capture settings of the camera to second cameracapture settings having a second resolution, wherein the secondcommunication channel is a different bandwidth channel than the firstcommunication channel, wherein the second resolution is different thanthe first resolution; and communicating second multimedia packets on thesecond communication channel during the video session, wherein thesecond multimedia packets correspond to the second images captured fromthe camera using the second camera capture settings having the secondresolution.

2. The method of any of the preceding paragraphs, wherein the mobiledevice communicates with a remote device, the method further comprising:determining that the remote device has transitioned from generatingmultimedia packets on the first communication channel to generatingmultimedia packets on the second communication channel during the videosession; in response to said determining that the remote device hastransitioned during the video session, changing the camera capturesettings of the camera to the second camera capture settings having thesecond resolution.

3. The method of any of the preceding paragraphs, wherein the mobiledevice uses an encoder, the method further comprising: in response tosaid changing the camera capture settings of the camera to the secondcamera capture settings having the second resolution, the mobile devicemodifying the encoder with different settings based on the changedresolution during the video session.

4. The method of any of the preceding paragraphs, wherein the mobiledevice communicates with a remote device; the method further comprising:after said modifying the encoder with different settings based on thechanged resolution, transmitting, by the mobile device, an intra frameon the second communication channel in response to said transitioning,wherein the intra frame transmitted by the modified encoder comprisesmetadata useable by the remote device to change settings of a videodecoder of the remote device.

5. The method of any of the preceding paragraphs, further comprising:determining that the intra frame transmitted on the second communicationchannel in response to said transitioning was not received by the remotedevice; retransmitting the intra frame on the second communicationchannel in response to said determining that the intra frame was notreceived by the remote device.

6. The method of any of the preceding paragraphs, further comprising:during a transition period of said modifying the encoder with differentsettings, ignoring refresh frame requests until the intra frametransmitted on the second communication channel is determined to havebeen received by the remote device.

7. The method of any of the preceding paragraphs, further comprising:during said communicating the first multimedia packets on the firstcommunication channel, maintaining packets in a first queue usable bythe video encoder; in response to the mobile device modifying theencoder with different settings, flushing the packets from the firstqueue.

8. The method of any of the preceding paragraphs, wherein the firstcommunication channel is a higher bandwidth channel than the secondcommunication channel, wherein the first resolution is a higherresolution than the second resolution.

9. The method of any of the preceding paragraphs, wherein the firstcommunication channel is WiFi, and wherein the second communicationchannel is cellular.

10. The method of any of the preceding paragraphs, wherein changing thecamera settings is performed in an asymmetric manner, with respect to aremote device.

11. A mobile device configured to perform a real-time application,wherein the real-time application communicates audio/video packets witha remote device, the mobile device comprising: a camera configured tocapture images for the real-time application; at least one antenna; atleast one radio, wherein the at least one radio is configured to performcommunication a first channel and a second channel; and one or moreprocessors coupled to the first radio, wherein the one or moreprocessors and the first radio are configured to: communicate firstmultimedia packets on a first communication channel during a videosession, wherein the first multimedia packets use first video settingsassociated with the first communication channel; in response totransitioning to communication of multimedia packets on a secondcommunication channel during the video session, change the first videosettings to second video settings for the second communication channel,wherein the second communication channel is a different bandwidthchannel than the first communication channel; and communicate secondmultimedia packets on the second communication channel during the videosession, wherein the second multimedia packets use the second videosettings associated with the second communication channel.

12. The mobile device of any of the preceding paragraphs, whereincommunication of the second multimedia packets on the secondcommunication channel is performed concurrently with communication ofthe first multimedia packets on the first communication channel.

13. The mobile device of any of the preceding paragraphs, wherein themobile device uses an encoder for the multimedia packets, wherein one ormore processors are further configured to: modify the encoder with thefirst or second video settings based on whether the mobile device iscurrently communicating on the first or second communication channel.

14. The mobile device of any of the preceding paragraphs, wherein thefirst communication channel is a higher bandwidth channel than thesecond communication channel, wherein the video encoder is configuredwith first settings for generation of higher resolution images when themobile device is communicating on the first communication channel, andwherein the video encoder is configured with second settings forgeneration of lower resolution images when the mobile device iscommunicating on the second communication channel.

15. The mobile device of any of the preceding paragraphs, wherein thefirst communication channel is a higher bandwidth channel than thesecond communication channel and wherein a lesser number of packets aretransmitted on the second communication channel than the firstcommunication channel.

16. The mobile device of any of the preceding paragraphs, wherein whenthe mobile device is communicating on the first communication channel,the mobile device is configured to generate a first number of videoframes on the first communication channel, wherein the first number ofvideo frames includes non-droppable frames and droppable frames, whereinwhen the mobile device is communicating on the second communicationchannel, the mobile device is configured to generate droppable framesand a lesser number of non-droppable frames on the second communicationchannel.

17. The mobile device of any of the preceding paragraphs, wherein thefirst communication channel is a higher bandwidth channel than thesecond communication channel, and wherein the first video settingsspecify a first resolution that is a higher resolution than a secondresolution specified by the second video settings.

18. The mobile device of any of the preceding paragraphs, wherein thefirst communication channel is WiFi, and wherein the secondcommunication channel is cellular.

19. The mobile device of any of the preceding paragraphs, wherein themobile device uses a first video encoder and a second video encoder,wherein the first video encoder is used for communication on the firstcommunication channel, and wherein the second video encoder is used forcommunication on the second communication channel, wherein the firstcommunication channel is a higher bandwidth channel than the secondcommunication channel, wherein the first video encoder is configuredwith the first video settings specifying generation of higher resolutionframes, and wherein the second video encoder is configured with thesecond video settings specifying generation of lower resolution frames.

20. A method for conducting a real time application on a mobile device,the method comprising: at the mobile device: capturing first images froma camera of the mobile device using first camera capture settings havinga first resolution; communicating first multimedia packets on a firstcommunication channel, wherein the first multimedia packets correspondto the first images captured from the camera using the first cameracapture settings having the first resolution; determining that theremote device has transitioned from generating multimedia packets on thefirst communication channel to generating multimedia packets on a secondcommunication channel, wherein the second communication channel is adifferent bandwidth channel than the first communication channel; inresponse to said determining that the remote device has transitioned,changing the camera capture settings of the camera to second cameracapture settings having a second resolution, wherein the secondresolution is different than the first resolution.

21. A method for conducting a real time application on a mobile device,the method comprising: at the mobile device: determining connectioncharacteristics of a first communication channel; selecting a firstvideo encoder from a plurality of possible video encoders based on thedetermined connection characteristics of the first communicationchannel; communicating multimedia packets on the first communicationchannel, wherein said communicating multimedia packets on the firstcommunication channel is performed using the first encoder of the mobiledevice; determining connection characteristics of a second communicationchannel; selecting a second video encoder from a plurality of possiblevideo encoders based on the determined connection characteristics of thesecond communication channel; communicating multimedia packets on thesecond communication channel, wherein said communicating multimediapackets on the first communication channel is performed using the secondencoder of the mobile device.

22. The method of any of the preceding paragraphs, wherein saidcommunicating multimedia packets on the first communication channel andsaid communicating multimedia packets on the second communicationchannel are performed for the same multimedia session (or are performedconcurrently)

Embodiments of the present invention may be realized in any of variousforms. For example, in some embodiments, the present invention may berealized as a computer-implemented method, a computer-readable memorymedium, or a computer system. In other embodiments, the presentinvention may be realized using one or more custom-designed hardwaredevices such as ASICs. In other embodiments, the present invention maybe realized using one or more programmable hardware elements such asFPGAs.

In some embodiments, a non-transitory computer-readable memory mediummay be configured so that it stores program instructions and/or data,where the program instructions, if executed by a computer system, causethe computer system to perform a method, e.g., any of a methodembodiments described herein, or, any combination of the methodembodiments described herein, or, any subset of any of the methodembodiments described herein, or, any combination of such subsets.

In some embodiments, a mobile device may be configured to include aprocessor (or a set of processors) and a memory medium, where the memorymedium stores program instructions, where the processor is configured toread and execute the program instructions from the memory medium, wherethe program instructions are executable to implement any of the variousmethod embodiments described herein (or, any combination of the methodembodiments described herein, or, any subset of any of the methodembodiments described herein, or, any combination of such subsets). Thedevice may be realized in any of various forms.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

What is claimed is:
 1. An apparatus for conducting a real timeapplication on a mobile device, the apparatus comprising: one or moreprocessing elements, wherein the one or more processing elements areconfigured to: capture first images from a camera of the mobile deviceas part of the real time application; communicate first multimediapackets on a first communication channel, wherein at least some of thefirst multimedia packets correspond to the first images captured fromthe camera, wherein the first communication channel is used forcommunication between the mobile device and a remote device, whereincommunicating the first multimedia packets on the first communicationchannel comprises using a first radio access technology (RAT), andwherein while communicating on the first communication channel, firstsettings are used for the real time application; capture second imagesfrom the camera of the mobile device as part of the real timeapplication; and communicate second multimedia packets on a secondcommunication channel using a second RAT, wherein at least some of thesecond multimedia packets correspond to the second images captured fromthe camera, wherein while communicating on the second communicationchannel, second settings are used for the real time application insteadof the first settings, wherein using the second settings instead of thefirst settings is performed based on using the second RAT.
 2. Theapparatus of claim 1, wherein communicating the first multimedia packetson the first communication channel comprises the mobile devicetransmitting and receiving the first multimedia packets over a firstlocal link that uses the first RAT, wherein communicating the secondmultimedia packets on the second communication channel comprises themobile device transmitting and receiving the second multimedia packetsover a second local link that uses the second RAT, wherein using thesecond settings instead of the first settings is performed based on themobile device communicating over the second local link that uses thesecond RAT.
 3. The apparatus of claim 1, wherein communicating the firstmultimedia packets on the first communication channel comprises theremote device transmitting and receiving the first multimedia packetsover a first remote link that uses the first RAT, wherein communicatingthe second multimedia packets on the second communication channelcomprises the remote device transmitting and receiving the secondmultimedia packets over a second remote link that uses the second RAT,wherein using the second settings instead of the first settings isperformed based on the remote device communicating over the second linkthat uses the second RAT.
 4. The apparatus of claim 1, wherein the firstsettings comprises first camera capture settings specifying a firstresolution, wherein said capturing the first images from the camera isperformed using the first camera capture settings, wherein the secondsettings comprises second camera capture settings specifying a secondresolution, wherein said capturing the second images from the camera isperformed using the second camera capture settings.
 5. The apparatus ofclaim 1, wherein the first settings comprise a first bit rate for thereal time application, wherein the second settings comprise a second bitrate for the real time application.
 6. The apparatus of claim 1, whereinthe one or more processing elements are further configured to: determineconnection conditions of the first communication channel; transition tousing the second communication channel instead of the firstcommunication channel based on determining the connection conditions ofthe first communication channel.
 7. The apparatus of claim 1, whereinthe one or more processing elements are further configured to: determineconnection conditions of the first communication channel; wherein usingthe second communication channel is performed based on determining theconnection conditions of the first communication channel and wherein thesecond multimedia packets are also communicated over the firstcommunication channel.
 8. The apparatus of claim 7, wherein the secondmultimedia packets comprise non-droppable frames and droppable frames,wherein the non-droppable frames are communicated over both the firstand second communication channels, and wherein a lesser number ofdroppable frames are communicated over the second communication channelthan the first communication channel.
 9. The apparatus of claim 7,wherein the one or more processing elements are further configured to:combine multimedia packets received over the first communication channeland the second communication channel to generate a video streamassociated with the real time application.
 10. The apparatus of claim 1,wherein communicating the second multimedia packets comprisescommunicating an intra frame comprising information identifying thesecond settings.
 11. A method for conducting a real time application ona mobile device, the method comprising: at the mobile device: capturingfirst images from a camera of the mobile device as part of the real timeapplication; communicating first multimedia packets on a firstcommunication channel, wherein at least some of the first multimediapackets correspond to the first images captured from the camera, whereinthe first communication channel is used for communication between themobile device and a remote device, wherein communicating the firstmultimedia packets on the first communication channel comprises using afirst radio access technology (RAT), and wherein while communicating onthe first communication channel, first settings are used for the realtime application; capturing second images from the camera of the mobiledevice as part of the real time application; and communicating secondmultimedia packets on a second communication channel using a second RAT,wherein at least some of the second multimedia packets correspond to thesecond images captured from the camera, wherein while communicating onthe second communication channel, second settings are used for the realtime application instead of the first settings, wherein using the secondsettings instead of the first settings is performed based on using thesecond RAT.
 12. The method of claim 11, wherein communicating the firstmultimedia packets on the first communication channel comprises theremote device transmitting and receiving the first multimedia packetsover a first remote link that uses the first RAT, wherein communicatingthe second multimedia packets on the second communication channelcomprises the remote device transmitting and receiving the secondmultimedia packets over a second remote link that uses the second RAT,wherein using the second settings instead of the first settings isperformed based on the remote device communicating over the second linkthat uses the second RAT.
 13. The method of claim 11, wherein the firstsettings comprises first camera capture settings specifying a firstresolution, wherein said capturing the first images from the camera isperformed using the first camera capture settings, wherein the secondsettings comprises second camera capture settings specifying a secondresolution, wherein said capturing the second images from the camera isperformed using the second camera capture settings.
 14. The method ofclaim 11, wherein the first settings comprise a first bit rate for thereal time application, wherein the second settings comprise a second bitrate for the real time application.
 15. The method of claim 11, furthercomprising: determining connection conditions of the first communicationchannel; wherein using the second communication channel is performedbased on determining the connection conditions of the firstcommunication channel and wherein the second multimedia packets are alsocommunicated over the first communication channel.
 16. The method ofclaim 15, wherein the second multimedia packets comprise non-droppableframes and droppable frames, wherein the non-droppable frames arecommunicated over both the first and second communication channels, andwherein a lesser number of droppable frames are communicated over thesecond communication channel than the first communication channel. 17.The method of claim 11, wherein communicating the second multimediapackets comprises communicating an intra frame comprising informationidentifying the second settings.
 18. A mobile device configured toperform a real-time application, the mobile device comprising: a cameraconfigured to capture images for the real-time application; at least oneradio, wherein the at least one radio is configured to performcommunication with the remote device over a first channel and a secondchannel; and one or more processors coupled to the at least one radio,wherein the one or more processors and the at least one radio areconfigured to: capture first images from the camera as part of the realtime application; communicate first multimedia packets on a firstcommunication channel using the at least one radio, wherein at leastsome of the first multimedia packets correspond to the first imagescaptured from the camera, wherein the first communication channel isused for communication between the mobile device and a remote device,wherein communicating the first multimedia packets on the firstcommunication channel comprises using a first radio access technology(RAT), and wherein while communicating on the first communicationchannel, first settings are used for the real time application; capturesecond images from the camera as part of the real time application; andcommunicate second multimedia packets on a second communication channelusing a second RAT, wherein at least some of the second multimediapackets correspond to the second images captured from the camera,wherein while communicating on the second communication channel, secondsettings are used for the real time application instead of the firstsettings, wherein using the second settings instead of the firstsettings is performed based on using the second RAT.
 19. The mobiledevice of claim 18, wherein communicating the first multimedia packetson the first communication channel comprises the remote devicetransmitting and receiving the first multimedia packets over a firstremote link that uses the first RAT, wherein communicating the secondmultimedia packets on the second communication channel comprises theremote device transmitting and receiving the second multimedia packetsover a second remote link that uses the second RAT, wherein using thesecond settings instead of the first settings is performed based on theremote device communicating over the second link that uses the secondRAT.
 20. The mobile device of claim 18, wherein the one or moreprocessing elements are further configured to: determine connectionconditions of the first communication channel; wherein using the secondcommunication channel is performed based on determining the connectionconditions of the first communication channel and wherein the secondmultimedia packets are also communicated over the first communicationchannel; wherein the second multimedia packets comprise non-droppableframes and droppable frames, wherein the non-droppable frames arecommunicated over both the first and second communication channels, andwherein a lesser number of droppable frames are communicated over thesecond communication channel than the first communication channel.